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As a diagnostic intervention (Bowles, Churchill, Effrat, & McDermott, 2002) for schools failing to meet school improvement goals, Ken¬ 
tucky used a scholastic audit process based on nine standards and 88 associated indicators called the Standards and Indicators for School 
Improvement (SISI). Schools are rated on a scale of 1-4 on each indicator, with a score of 3 considered as fully functional (Kentucky De¬ 
partment of Education [KDE], 2002). As part of enacting the legislation, KDE was required to also audit a random sample of schools that 
did meet school improvement goals; thereby identifying practices present in improving schools that are not present in those failing to im¬ 
prove. These practices were referred to as variance points, and were reported to school leaders annually. Variance points have differed from 
year to year, and the methodology used by KDE was unclear. Moreover, variance points were reported for all schools without differentiating 
based upon the level of school (elementary, middle, or high). In this study, we established a transparent methodology for variance point 
determination that differentiates between elementary, middle, and high schools. 
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Introduction 

The Standards and Indicators for School Improvement 
(SISI) were put in place in Kentucky’s public schools 
in 2000 to support the Scholastic Audit process, which 
was enacted in 1998 as part of the school accountabil¬ 
ity legislation (703 Kentucky Administrative Regula¬ 
tion 5:120). These nine standards and 88 indicators 
were the basis for scholastic audits of schools failing to 
meet school improvement goals, as well as a sample of 
how schools were meeting goals. Audits of these two 
groups of schools with the SISI revealed factors that 
distinguished the groups, and that was viewed as valu¬ 
able in focusing the improvement efforts of schools 
and school districts across Kentucky. KDE initially re¬ 
ferred to these indicators as leverage points , but 


changed the reference to variance points because these 
indicators represented variance in audit results be¬ 
tween the two groups of schools. KDE communicated 
these variance points to school leaders as information 
to assist in prioritizing school improvement practices 
(Kentucky Department of Education [KDE], 2003). It 
is the identification and subsequent use of identified 
indicators that form the foundation for the variance 
point concept and provide the basis for this research. 

The variance point concept presented great intui¬ 
tive appeal and practical application for school leaders 
in Kentucky. It stood to reason that if a discrete set of 
best practices were demonstrated empirically as true 
variance points, the prioritization of school improve¬ 
ment efforts would become clearer. New variance 
point reports were issued after each cycle of audits, 
with results ranging widely in terms of the number 
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and emphasis of the indicators that were identified as 
variance points. For example, there were 11 variance 
points reported for the 2004 scholastic audit cycle 
(Common Variance Points 2004, n.d.), yet the 2004- 
2005 report identified 51 variance points (Variance 
Points 2004-2005, n.d.). Moreover, an investigation of 
perceptions of the impact of the SISI on school im¬ 
provement revealed that elementary school personnel 
had greater confidence in, and agreement with, SISI 
recommendations than middle or high school person¬ 
nel (Appalachian Educational Laboratory [AEL], 
2002 ). 

Statement of the Problem 

The SISI and related variance points were a valuable 
tool for school leaders involved in school improve¬ 
ment, yet inconsistencies in variance point results and 
differences in perceptions between elementary and 
middle/high school personnel as to the efficacy of the 
process suggested a need to report the variance points 
by school level. The purpose of this study was to apply 
appropriate measures of association to scholastic audit 
results for schools classified as Assistance Level 3 and 
Meeting Goals for the period from 2004-2008 to iden¬ 
tify significant school improvement indicators for ele¬ 
mentary, middle, and high schools. 

Background 

An understanding of variance points was predicated 
upon an understanding of the SISI and Kentucky's 
accountability system. The SISI were only a part of a 
large, complex system of assessments, criteria, sanc¬ 
tions, and supports stipulated by the Kentucky Gen¬ 
eral Assembly for schools failing to meet improvement 
goals (Kentucky Revised Statutes 158.6455(5)). 
Bowles, Churchill, Effrat, and McDermott (2002) de¬ 
veloped the Intervention Decision-Making Framework 
for use by policymakers to provide clarity as to how 
the elements of education accountability systems inter¬ 
related. This framework was composed of: Perform¬ 
ance Criteria, Strategic Criteria, Diagnostic Interven¬ 
tion, Corrective Intervention, Targets, Tactics, and Exit 
Criteria. 


Kentucky's Educational 
Accountability Model 

The Commonwealth Accountability and Testing Sys¬ 
tem (CATS) classified schools as Assistance Level, Pro¬ 
gressing, or Meeting Goals based upon school per¬ 
formance relative to improvement goals. KDE was the 
agency responsible for the administration of CATS and 
related interventions. 

Performance criteria. Criteria for each classification 
related to the accountability index for each school, a 
score from 0 to 140, based upon student performance 
on a combination of academic and nonacademic indi¬ 
cators. The statewide goal for schools in 2014 was an 
accountability index of 100. The baseline accountabil¬ 
ity index for each school was determined based upon 
2000 CATS results and was used to determine the goal 
line leading from the baseline to the goal of 100 in 
2014 (Bowles et ah, 2002; 703 KAR 5:020; KDE, 
2003). 

By regulation (703 Kentucky Administrative Regu¬ 
lation 5:020), schools were classified each biennium as 
Meeting Goals, Progressing, or Assistance based upon 
whether the average accountability index for that bi¬ 
ennium exceeded the goal line (Meeting Goals), fell 
below the assistance line (Assistance) or fell between 
the goal line and assistance line (Progressing). Figure 1 
(page 3) illustrates the interaction between the ac¬ 
countability index and the goal line, thereby determin¬ 
ing the level for each school. 

Strategic criteria. To facilitate the allocation of the 
most intensive interventions to the schools in the most 
need, schools classified as Assistance were further di¬ 
vided into three equal groups or assistance levels 
(Level 1, Level 2, and Level 3) based upon account¬ 
ability indices. Level 3 Assistance schools made up the 
lowest-performing third of the assistance schools 
(Bowles et ah, 2002; KDE, 2003). 

Diagnostic Criteria. Scholastic audits were re¬ 
quired of both Assistance Level 2 and Assistance Level 
3 schools (Bowles et ah, 2002). Scholastic audits used 
the SISI as the basis for the evaluation of three areas of 
school functioning: academic performance, learning 
environment, and efficiency. As described in Appendix 
A, these three areas encompassed nine standards, 
composed of 88 indicators (KDE, 2002). 
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Each of the 88 indicators was supported by a scor¬ 
ing rubric that specified the types of evidence to be 
examined, as well as descriptors of the evi¬ 
dence required for each of four categories of classifica¬ 
tion: (a) Rating of 1—little or no development or im¬ 
plementation; (b) Rating of 2—limited development or 


partial implementation; (c) Rating of 3—fully func¬ 
tioning and operational level of development and im¬ 
plementation; and (d) Rating of 4—exemplary level of 
development and implementation. A rating of 3 
was viewed as on-target, and a score of 4 as ideal 
(KDE, 2002). 


Figure 1. Key Components of Kentucky's Long-Term Education Accountability Model 



Corrective intervention. The results of the scholastic 
audit guided the nature of school improvement activi¬ 
ties, which were led in part by a Highly Skilled Educa¬ 
tor (HSE) who was assigned to the school by KDE 
(703 KAR 5:120; Bowles et ah, 2002). 

Target and tactic. School improvement interventions 
included professional development for teachers and 
school administrators as appropriate and as informed 
by the Scholastic Audit. These interventions were 
monitored by the HSE, whose primary goal was to 
facilitate change (Bowles et ah, 2002). 

Exit criteria. Scholastic audit recommendations guided 
school improvement efforts and served as criteria for 
schools to exit assistance (Bowles et al., 2002). 

The Scholastic Audit Process 

In 1998, the Kentucky General Assembly enacted leg¬ 
islation requiring an audit of all the schools failing to 
meet state improvement goals, and of a sample of 


schools successful in meeting state improvement goals. 
A team composed of at least one HSE, one parent, one 
teacher, one administrator and one university faculty 
member conducted the audits. The audit team ad¬ 
dressed the learning environment and efficiency of the 
school, the academic performance of the students, and 
the school council's data analysis and planning prac¬ 
tices. Recommendations were to be made by the team 
to the Kentucky Board of Education (KBE) regarding 
the school's performance classification and the assis¬ 
tance required to address deficiencies at each 
school (KRS 158.6455[5]). 

The SIS I 

In 2000, KBE adopted administrative regulations that 
identified guidelines for these scholastic audits, and 
specified the SISI as the basis of the audits (703 KAR 
5:120). Best practices were divided into three do¬ 
mains: (a) Academic Performance, (b) Learning Envi- 
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ronment, and (c) Efficiency. There were three stan¬ 
dards associated with each of these areas: (a) Academic 
Performance was composed of Curriculum (Standard 
1), Classroom Evaluation/Assessment (Standard 2), 
and Instruction (Standard 3); (b) Learning Environ¬ 
ment was composed of School Culture (Standard 4), 
Student, Family, and Community Support (Standard 
5), and Professional Growth, Development, and 
Evaluation (Standard 6); and (c) Efficiency was com¬ 
posed of Leadership (Standard 7), Organizational 
Structure and Resources (Standard 8) and Comprehen¬ 
sive and Effective Planning (Standard 9) (KDE, 2002). 

The purpose behind the SISI is not unlike the ef¬ 
forts behind the work of others who have sought to 
identify practices that are common in schools that 
show a high level of student learning. In response to 
the Equality of Educational Opportunity Study (Cole¬ 
man et. al, 1966) researchers such as Brookover and 
Lezotte (1979) and Edmonds (1981) identified prac¬ 
tices prevalent in high-poverty, high-achieving schools. 
Their work, along with the work of other effective 
school researchers, became the foundation for the Ef¬ 
fective Schools Correlates. More recently, researchers 
such as Marzano (2003), Reeves (2003), and McEwan 
(2008) have identified practices in schools that reflect 
a high level of student learning. A number of practices, 
correlates, or scholastic indicators emerge that, when 
applied, have been shown to improve student learn¬ 
ing. 

Variance Points 

The initial legislation stated that, for "informational 
purposes," schools meeting state goals were to be 
audited and the results of these audits reported (KRS 
158.6455 [5]). Regulations specified that a randomly 
selected sample of schools meeting goals were to be 
audited, but did not specify the parameters of the 
audit (703 KAR 5:120). As KDE completed full audits 
of Level 3 Assistance schools and a sample of Meeting 
Goals schools, comparisons were made between the 
audit results of these two groups of schools. In 
2003, KDE published an analysis of these comparisons 
that identified a subset of the 88 indicators as related 
to school improvement and referred to these as vari¬ 
ance points. Based upon the 2002-2003 audit data, 27 
of the 88 indicators were identified as variance points 
and used as a basis for best practice recommendations 


for Kentucky's schools. It was unclear what method 
was used to determine these original variance points 
(KDE, 2003). 

Since the publication of the 2002-2003 variance 
points document, additional reports were created 
based upon scholastic audits for 2004, 2004-2005, 
and 2004-2006 (Common Variance Points 2004, n.d.; 
Variance Points 2004-2005, n.d.; Variance Points 
2004-2006, n.d.). The number of variance points re¬ 
ported ranged from 11 to 51. 

Diagnostic Nature of the SISI and Vari¬ 
ance Points 

The conceptual framework established by Bowles et al. 
(2002) identified the role of the SISI as that of a diag¬ 
nostic intervention. Information based on the SISI di¬ 
rectly influenced the corrective interventions and exit 
criteria for schools in Assistance Level 2 and Assistance 
Level 3 (Bowles, et al., 2002). Variance points served 
the purpose of highlighting promising practices for all 
schools in Kentucky (KDE, 2003). If the SISI effec¬ 
tively diagnosed problems within struggling schools, 
then by definition, variance points communicated to 
school leaders specific policies, practices, and charac¬ 
teristics that led to school improvement in Kentucky. It 
was critical that variance point information be correct 
and appropriate for all schools, regardless of school 
level. Confidence in the scholastic audit process and 
the SISI was lower for secondary personnel than ele¬ 
mentary personnel (AEL, 2002), pointing to either a 
lack of understanding of the SISI or recommendations 
that were not reflective of the school environment in 
question. 

It is important to note that much of the research 
examining effective school practices has looked more 
at the schooling process across P-12 than at effective 
practices found within elementary, middle, or high 
schools. Certainly child development is different at 
each level. Eligh schools tend to be larger and more 
departmentalized by content area than do elementary 
schools. The needs of middle school students differ 
from that of their younger and older counterparts. 
Therefore, it seems likely that not all educational 
strategies and instructional practices would be equally 
effective at each level. Eleretofore, the research com¬ 
paring the effectiveness of identified practices when 
implemented at each school level has been limited. 
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Statement of Purpose 

The purpose of this study was to provide school lead¬ 
ers serving at each level—elementary, middle, and 
high school—with an empirically based listing of indi¬ 
cators from the SISI that were statistically significantly 
related to schools meeting goals, as compared to those 
in Assistance Level 3. Significant indicators were de¬ 
termined using scholastic audit results from 2004 to 
2008, calculating elementary, middle, and high school 
data separately. Results of these school-level calcula¬ 
tions were compared to each other and to current vari¬ 
ance points from KDE to answer the following ques¬ 
tions: 

1. In what standards did significant indicators 
occur for elementary, middle, and high 
schools? How did these compare to the extant 
KDE variance points? 

2. What types of significant indicators were 
common across all three levels of schools? 
What types of indicators were significant for 
specific school levels only? 

3. Which indicators were the most related to 
school improvement for each school level? 

4. Were there indicators that were not demon¬ 
strated to be significant for any level of school? 
What is the implication for these indicators? 

5. Based upon most related indicators, what as¬ 
sociated best practice was suggested? 

Variables 

Since the sample was a purposive, convenience sam¬ 
ple, demographic and accountability measures were 
used to compare the nature of the Level 3 Assistance 
and Meeting Goals schools. 

Poverty level of school. School poverty levels were 
estimated using the 2005-2006 free and reduced 
lunch participation rates for each school (Nutrition 
and Health Services, 2005). 

Community type. The nature of each community 
in terms of urban-rural character was estimated using 
the Urban-Rural Continuum from the United States 
Department of Agriculture (USDA) for 2003 for the 
county in which the school resided. The Urban-Rural 
Continuum Codes ranged on a scale from 1-9, based 
on total population, and population density. For this 
study, ratings were grouped as (a) Metro, ratings 1,2, 


or 3; (b) Urban-metro adjacent, ratings of 4 or 6; (c) 
Urban, ratings of 5 or 7; (d) Rural-metro adjacent, a 
rating of 8; and (e) Rural, a rating of 9. 

Academic index. The Academic Index for each 
school was used to describe student achievement at 
the time of the audit. This information was included 
with the data set provided to researchers from KDE. 

Method 

Participants 

A purposive, convenience sample of 60 Kentucky ele¬ 
mentary, middle, and high schools was used. During 
the period from 2004 to 2008, scholastic audits were 
performed with these schools either because they were 
classified as Level 3 Assistance or chosen from schools 
Meeting Goals. Of these schools, 24 were classified as 
Level 3 Assistance and 36 as Meeting Goals. Of the 
Level 3 Assistance schools, there were seven elemen¬ 
tary, 10 middle, and seven high schools. Meeting Goals 
schools were composed of 19 elementary, eight mid¬ 
dle, and nine high schools. 

Sample description. Academic and socioeconomic 
indicators for the period from 2004 to 2008 were 
summarized for the sample in Table 1 (page 4). Level 3 
Assistance schools achieved at a lower level than did 
schools classified as Meeting Goals that were audited, 
which would be expected given the criteria for group 
membership. Socioeconomic measures indicated that 
Level 3 Assistance schools were poorer and less rural 
than schools classified as Meeting Goals that were 
audited. 

Instrumen tation 

The Kentucky Department of Education developed the 
SISI for the purpose of conducting audits required for 
Assistance Level and other selected schools. The indi¬ 
cators and related criteria were reportedly derived 
from a published research base. Criteria for each of the 
88 indicators was articulated through rubrics (KDE, 
2006), which were applied by the scholastic audit 
team (KDE, 2002). These teams were trained on the 
use of the rubrics by the KDE (2002) to estab¬ 
lish consistency between audits. There were no pub¬ 
lished reports of the construct validity or reliability for 
the indicators, the rubric criteria, or the audit process. 
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Table 1. Sample Characteristics by School Level and Accountability Classification 


Academic and demographic 
descriptors 

Elementary 

AL3 MG 

AL3 

Middle 

MG 

AL3 

High 

MG 

Academic Index (Mean) 

57.2 

86.1 

53 

82.4 

57.8 

82.3 

Free & Reduced Lunch 

(Mean) 

89.3% 

60.5% 

79.8% 

37.8% 

64.6% 

30.2% 

Community Type 

Metro 

4 

4 

9 

3 

4 

3 

Urban-Metro Adjacent 

1 

4 

0 

3 

0 

2 

Urban 

2 

6 

0 

2 

1 

4 

Rural-Metro Adjacent 

0 

2 

1 

0 

1 

0 

Rural 

0 

3 

0 

0 

1 

0 


Analysis 

The Kendal-Stuart Tau-c was the measure of associa¬ 
tion used with ranked, ordinal data (Mehta & Patel, 
1996). Scholastic Audit data for each indicator was 
represented by a number from 1-4, with 1 represent¬ 
ing little or no implementation, and 4 representing 
exemplary implementation. The Kendal-Stuart Tau-c 
represented as a number between -1 and 1 the 
strength of relationship between these indicator ratings 
and the school classification (Level 3 Assistance, Meet¬ 
ing Goals). A Tau-c was calculated using SPSS vl3.0 
for each of the 88 indicators, for each of the three 
school levels resulting in 264 calculations. Multiple 
comparisons increased Type-I error and required the 
use of a p = 0.000 for statistical significance. 

Procedure 

Ratings for all scholastic audits (N=296) conducted 
from 2002 through the spring of 2008 were provided 
to researchers by the KDE in an electronic format. This 
file contained the ratings (1 to 4) for each audited 
school on each of the 88 indicators, the performance 
level for each school at the time of the audit (Assis¬ 
tance Level 3, Assistance Level 2, Meeting Goals, etc.), 
the year of the audit, and the academic index for the 
school at the time of the audit. A purposive, conven¬ 
ience sample was selected for the study that met the 
following conditions: (a) the audit was con¬ 
ducted since January 2004; (b) the audit was for a 
school classified as either Level 3 Assistance school or 


Meeting Goals; (c) the school was not a P-12 school; 
and (d) if schools were audited more than once since 
2004, only the oldest audit was included in the 
study. Sixty schools met the stated criteria for use in 
the study. 

Schools in the sample were coded in terms of 
classification (Assistance Level 3, Meeting Goals) and 
level (elementary, middle, or high). SPSS vl3.0 was 
used to determine the Kendall-Stuart Tau-c for each 
indicator at each level. Once all analyses were com¬ 
plete, significant indicators were identified for each 
level of school. 

Results 

This study sought to refine the variance point concept 
by disaggregating audit data based upon school level 
(elementary, middle, and high), thereby providing 
school leaders with more precise guidance for school 
improvement in Kentucky. An analysis of the results 
does provide some guidance for leadership decisions, 
based on the level of the school. 

In what standards did the variance points 
occur? How did these compare to KDE 
variance points? 

Analysis of audit data revealed 21 significant indicators 
for elementary schools, 14 significant indicators for 
middle schools, and 33 significant indicators for high 
schools. Significant indicators were examined relative 
to the standards they were contained within, and 
also to the 43 variance points reported by KDE for the 
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same period. Variance points from KDE and significant 
indicators for elementary, middle, and high schools 
were distributed across the standards, as indicated in 
Tables 2, 3, and 4 (page 6). Differences existed be¬ 
tween the standards of emphasis between all three lev¬ 


els of school and also with KDE variance points. Eight 
of the 43 variance points reported by KDE were not 
significant for any of the three school levels: 2.1e, 2.If, 
4.Id, 4.1k, 6.1a, 6.1b, 7.1c, and 8.1c. (See appendix 
for a description of each indicator.) 


Table 2. Significant Indicators from the Academic Performance Standards Compared by School Level and with 
KDE Variance Points from 2004-2007 
Academic Performance Standards and Indicators 


Curriculum Classroom Evaluation and Instruction 

Assessment 


SISI 

KDE 

E 

M 

H SISI 

KDE 

E 

M 

H 

SISI 

KDE 

E M 

H 

1.1a 

X 

X 


2.1a 



n/a 


3.1a 

X 

X 

X 

1.1b 




2.1b 

X 


n/a 

X 

3.1b 




1.1c 




2.1c 

X 

X 

n/a 


3.1c 



X 

l.ld 




2.Id 

X 



X 

3.Id 

X 

X 

X 

l.le 


X 


2.1e 

X 




3.1e 


na 


l.lf 




2.If 

X 




3.If 

X 


X 

llg 

X 

X 


2-lg 





3-lg 








2.1h 


X 



3.lh 

X 

X 



All indicators significant at the p =.000 level, n/a denotes no rating of 3 or 4 reported in the sample. 


Elementary schools. The 21 significant indicators for 
elementary school were distributed across eight of the 
nine standard areas; only Standard 3 did not have a 
significant elementary indicator. Compared to the ex¬ 
tant KDE variance points, the elementary indicators 


were common to 16 indicators, with five new indica¬ 
tors emerging from the disaggregated analysis. These 
emerging indicators included: l.le, 2.1h, 5.1b, 9.3b, 
and 9.4b. 


Table 3. Significant Indicators from the Learning Environment Standards Compared by School Level and with 
KDE Variance Points from 2004-2007 
Learning Environment Standards and Indicators 


School Culture Student, Family and Community Professional Growth, Develop- 

Support ment and Evaluation 


SISI 

KDE 

E 

M 

H 

SISI 

KDE 

E 

M 

H 

SISI 

KDE 

E 

M 

H 

4.1a 

X 


X 

X 

5.1a 

X 

X 


X 

6.1a 

X 




4.1b 

X 



X 

5.1b 


X 


n/a 

6.1b 

X 




4.1c 

X 

X 

X 

X 

5.1c 

X 



X 

6.1c 



X 

X 

4.Id 

X 




5.Id 

X 

X 

X 

X 

6.Id 





4.1e 

X 


X 


5.1e 



X 


6.1e 

X 



X 

4.If 




n/a 






6.If 




X 

4-lg 

X 


X 

X 






6.2a 





4.1h 

X 

X 

X 

X 






6.2b 

X 



X 

4. li 

X 


X 

X 






6.2c 

X 

X 

X 


4-lj 

X 

X 


X 






6.2d 

X 

X 



4.1k 

X 









6.2e 















6.2f 

X 

X 


X 


All indicators significant at the p=.000 level, n/a denotes no rating of 3 or 4 reported in the sample 
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Middle schools. There were only 14 indicators that 
were significant for middle schools, the least amongst 
the school levels. These were divided across only five 
of the nine standards, with Standards 1, 2, 7, and 9 
absent significant indicators. Compared to the extant 


KDE variance points, the middle school indicators 
were common to 12 indicators, with two indicators 
emerging from the disaggregated analysis. These 
emerging indicators included: 5.1e and 6.1c. 


Table 4. Significant Indicators from the Efficiency Standards Compared by School Level and with KDE Variance 
Points from 2004-2007 
Efficiency Standards and Indicators 

Leadership Organizational Structure and Comprehensive and Effective 

Resources Planning 


SISI 

KDE 

E 

M 

H 

SISI 

KDE 

E 

M 

H 

SISI 

KDE 

E 

M 

H 

7.1a 




X 

8.1a 





9.1a 



na 


7.1b 

X 

X 


X 

8.1b 

X 

X 



9.2a 





7.1c 

X 




8.1c 

X 




9.2b 





7.Id 





8.Id 

X 

X 



9.3a 





7.1e 

X 



X 

8.1e 




X 

9.3b 


X 



7.If 

X 

X 


X 

8.If 

X 




9.3c 




X 

7-lg 

X 



X 

8.2a 




X 

9.4a 





7.1h 

X 

X 


X 

8.2b 





9.4b 


X 



7. li 



na 


8.2c 





9.5a 




X 

7- lj 




X 

8.2d 

X 


X 


9.5b 





7.1k 

X 



X 






9.5c 



na 



9.5d 

9.6a na 

9.6b 

9.6c 

9.6d na 

All indicators significant at the p=0.000 level, n/a denotes no rating of 3 or 4 reported in the sample. 


High schools. There were 33 significant indicators 
for high schools, the most of the three school levels. 
Standard 1 did not contain any significant indicators 
with the other indicators shared by the other eight 
standards. Compared to the extant KDE variance 
points, the high school indicators were common to 24 
indicators, with nine new indicators emerging from the 
disaggregated analysis. These emerging indica¬ 
tors included: 3.1c, 6.1c, 6.If, 7.1a, 7.lj, 8.1e, 8.2a, 
9.3c, and 9.5a. 

What indicators were common across all 
school levels? Which were unique 
to certain school levels? 

There were three indicators that were commonly sig¬ 
nificant for elementary, middle, and high schools. 


These indicators were part of the KDE variance points 
and were from Standards 4 and 5: 4.1c, 4.1h, and 
5.Id. There were 11 significant indicators unique to 
elementary schools, four unique to middle schools, 
and 17 unique to high schools. Six indicators were 
significant for middle and high schools only, and one 
for elementary and middle schools. These indica¬ 
tors were considered as secondary and elementary, 
respectively. Secondary-only indicators included: 3.1a, 
3.Id, 4.1a, 4.1g, 4.li, and 6.1c. Only 6.2c was identi¬ 
fied as an elementary-only indicator. 

Which indicators were most related to 
school improvement? 

As the absolute values of the Kendall-Stuart Tau-c in¬ 
creased, the strength of relationship between each in- 
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dicator and the school improvement classification 
(AL3, MG) increased. Table 2 highlighted the indica¬ 
tors by level of strength for each school level. To aid 
school leaders in prioritizing school improvement ef¬ 
forts, significant indicators were leveled into tiers 
composed of approximately five indicators each. Tier 1 
indicators for all three levels of schools in¬ 
cluded Standard 4 indicators, with indicator 4.1h as 
the only indicator in Tier 1 shared by all three levels. 
At the elementary level, Standard 2 indicators com¬ 
plemented Standard 4 indicators, whereas the middle 


school was almost entirely Standard 4, except for 
Standard 8.Id, which was related to the efficient use of 
time by staff. Tier 1 at the high school level included 
indicators related to leadership (7. lg), assessment 
(2.1b), teacher content knowledge (3.Id), and the 
availability of instructional assistance for students out¬ 
side of class (5.Id). Tier 2 and Tier 3 became increas¬ 
ingly divergent, as no indicators were shared across 
school levels. 


Table 5. Tiered SISI Indicators by School Level with Tc Coefficients 



Elementary 

Middle 

High 

Tier 1 

.800 <= Tc < to 1.0 

None 

8.Id 

4.1a, 4.1h, 7.1g, 

Tier 2 

,700<=Tc<.800 

4.1c 

4.1c, 4.1e, 4.1g, 4.1h 

2.1b, 3.Id, 3.If, 4.1b, 

4. lj, 5.Id, 6.1c, 6.2b, 
6.2f, 7.1h, 9.3c 

Tier 3 

.600<=Tc<.700 

l.lg, 2.1c, 2.1h, 4.1h, 

4. lj, 5.1a, 6.2d, 7.1f, 

3.1h, 4.1a, 4.1i, 5.1e 

2.Id, 3.1a, 3.1c, 4.1c, 
4.1g, 4.li, 5.1c, 6.1e, 

6.If, 7.1a, 7.1b, 7.1e, 

7.If, 7.lj, 7.1k, 8.1e 

Tier 4 

.500<=Tc<.600 

5.1b, 5.Id, 6.2c, 6.2f, 
7.1b, 7.1h, 8.1b, 8.Id, 
9.3b 

5.Id, 6.1c, 6.2c, 



All values significant at p=.000 


Which indicators were not related to 
school improvement? 

There were 40 indicators that were not significantly 
related to the school improvement designations. Of 
these, 12 were from Standard 9, six from Standard 
8, five from Standard 6, four each for Standards 1 and 

2, and three each for Standards 3, 4, and 7. All indica¬ 
tors in Standard 5 were significant to at least one level 
of school. These indicators were denoted in Tables 2, 

3, and 4. 

Discussion 

The standards and indicators for School Improvement 
are clearly pivotal to school accountability and im¬ 
provement in Kentucky. The SISI provides for diagnos¬ 
tic intervention and establishes the framework for im¬ 
provement activities in Assistance schools. Moreover, 


lessons learned through the application of the SISI and 
the exploration of the variance point concept affect 
schools in the state regardless of accountability classifi¬ 
cation. As school leaders work to create focused im¬ 
provement by prioritizing problems and solutions, it 
becomes important for all that the information pro¬ 
vided regarding best practices through the variance 
points is valid and reliable. 

As significant indicators and variance points are 
discussed, it is important to recognize that the variance 
point concept identifies gaps in practice—elements 
that are present in one setting but not in another. 
When an indicator is not significant or termed a vari¬ 
ance point, that does not mean that it is unimportant; 
rather, it may be fundamental and therefore inherently 
present in both settings. For example, indicator 1.If 
states that "there is in place a systematic process for 
monitoring, evaluating, and reviewing the curricu¬ 
lum." This practice was not found to differ signifi- 
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cantly in Assistance Level 3 or Meeting Goals schools 
in this analysis, nor was it a variance point in the KDE 
2004-2007 report. Yet, this is an important indicator, 
a practice that would be considered as fundamental in 
school today. 

Common significant indicators. Only three indicators 
for school improvement were significant for elemen¬ 
tary, middle, and high schools. This reinforces the per¬ 
ception by survey respondents in 2002 (AEL, 2002) 
with regard to the efficacy of audit recommendation. 
This finding implies that, although a very few practices 
made a significant impact for all levels of schools, the 
impact of specific school improvement practices may 
differ between elementary, middle, and high schools. 
This idea has implications not only for practitioners 
serving at the various school levels (elementary, mid¬ 
dle, and high) but also for those who train aspiring 
school leaders and those who provide their profes¬ 
sional development. Some educational practices seem 
to have varying degrees of impact, depending on the 
level of the school. 

The three common significant indicators are (a) 
4.1c: teachers hold high expectations for all students 
academically and behaviorally and this is evidenced in 
their practice; (b) 4.1h: there is evidence that the 
teachers and staff care about students and inspire their 
best efforts; and (c) 5.Id: students are provided with a 
variety of opportunities to receive additional assistance 
to support their learning, beyond the initial classroom 
instruction. Indicators 4.1c and 4.1h clearly imply that 
hiring practices made a difference between the Assis¬ 
tance and Meeting Goals schools, as at the root, these 
indicators are dispositional. This finding additionally 
validates discussions among teacher education units 
with regard to the measurement of dispositions. Indi¬ 
cator 5.Id is timely because differentiated instructional 
models are currently being developed in many schools, 
validating this practice to school leaders who are 
working hard to develop capacities in instruction and 
assessment to meet the needs of all students. 

Extant KDE variance points that were not congru¬ 
ent with the significant indicators in this study differed 
in nature. Again, these practices are not unimportant, 
rather simply were not significantly different between 
the two groups of schools. For example, the involve¬ 
ment of teachers and nonteaching staff in decision 
making as an indicator does reflect an accepted prac¬ 
tice. However, it is conceivable that this practice might 


occur sufficiently in all settings through school-based 
decision making (SBDM), required in Kentucky, 
thereby not yielding statistically significant different 
audit ratings. There are several indicators that, for 
middle and high school, there was no rating higher 
than a 2, with a 3 representing target. These findings 
suggest that there may be untapped potential with re¬ 
gard to the implementation of some practices. If a 
school can meet goals with a rating of 1 or 2, how 
much better could things be if the school fully func¬ 
tioned in the indicator area? The indicators in question 
addressed the frequency, rigor, and authenticity of 
classroom assessments in the middle schools, the use 
of multiple assessments, and the access of students to 
guidance and other supports in the high schools. 
Council policies and planning in general were cause 
for a rating of 1 or 2 in the middle school. School 
leaders in schools that are Progressing or in Meeting 
Goals schools that have ceased improving may find 
that these areas offer opportunities for growth for the 
school. 

School culture. The relative strength of relationship of 
indicators suggests that Standard 4, School Culture, is 
key to differences between the schools in the sample. 
This finding is consistent conceptually with the KDE 
variance points, although the ordering of specific indi¬ 
cators may vary. Standard 7, which addresses leader¬ 
ship, was more prominent in high school results than 
in elementary and middle schools. This may be due to 
the complex nature of high schools, which require 
more from school leaders. 

Standard 9. The limited number of variance points for 
Standard 9 (none identified by KDE and two at the 
elementary and two at the high school identified by 
this study) may call into question the impact this stan¬ 
dard and related indicators have on student achieve¬ 
ment. While short- and long-range planning have been 
an expected activity for school leaders, the difference 
in the ways educators in Meeting Goals schools and 
Level 3 schools plan was not statistically different. The 
schools’ improvement plans seem to have a minimal 
effect in practice, suggesting a review of how im¬ 
provement plans are developed and used to affect stu¬ 
dent achievement. 

High schools. CATS data reveal that, as a whole, ele¬ 
mentary schools consistently outpace middle and high 
schools (KDE, 2007). Yet a portion of this data analysis 
reveals 30 practices (variance points) that are com- 
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pleted more effectively at Meeting Goals high schools 
than at Level 3 schools. High school leaders and as¬ 
signed HSEs may want to consider a more focused 
approach to student achievement, tapping into the 
variance points identified specifically for high schools. 
Additionally consideration may be given to eliminat¬ 
ing practices that have little or no empirical evidence 
of improving student achievement. This more focused 
approach should improve student learning and pro¬ 
vide additional time to engage in practices associated 
with identified variance points. 

Limitations and Recommendations 

Researchers found the sample of Assistance and Meet¬ 
ing Goals schools to be very different demographically. 
Assistance schools typically were poorer, with many 
located in metropolitan communities, whereas the 
Meeting Goals schools were less impoverished and 
much more rurally located. There are poor, metropoli¬ 
tan schools in Kentucky that are Meeting Goals; they 
were just not audited. Researchers suggest a matched- 
pairs approach to selecting Meeting Goals schools 
would provide more valid data with which to draw 
conclusions concerning variance points. 

Sample size was also an issue, as the small sample 
limited the choice of statistical processes that work 
with the data provided. As more and more schools are 
audited, a sufficient sample size will accumulate to 
support more interpretable statistics, such as Chi- 
square, that were not appropriate given the nature of 
the data at this time. 

Consistency of ratings between audit teams needs 
further discussion. While the training of audit team 
members involves multiple days of instruction, the 
lack of activities to ensure inter-rater reliability weak¬ 
ens comparison between schools audited by different 
teams. Clearly delineated practices under each rating 
level and suggested examples of supporting evidence 
tend to guide team rankings. Nevertheless, measures 
of consistency of rankings between audit team mem¬ 
bers and audit teams warrants review. Since the results 
from these audits are used to shape policy even in the 
absence of such validation studies, the assumption was 
made that the ratings were consistent for the purpose 
of this analysis. 

As 2014 fast approaches, the date by which all 
Kentucky schools are to be measured as proficient, the 


need to tailor educational practices to the schools and 
the students they serve grows. While this study sug¬ 
gests that the effectiveness of certain educational prac¬ 
tices varies from elementary to middle to high schools, 
the same may also hold true related to other demo¬ 
graphic factors. For example, practices that have a sig¬ 
nificant, positive impact on student learning in large, 
urban high schools may have less of an impact on 
large, rural high schools. Achievement gaps among 
various student populations within schools also sup¬ 
port the notion that not only do instructional practices 
need to be modified to meet the varying needs of the 
students, but leadership practices may need review as 
well. Finally, some may embrace the SISI as a “one size 
fits all” type of leadership and educational practice. In 
reality, the nuances found among students within a 
school and from one school to another clearly call for 
more measured, data-informed educational decisions 
and practices. 
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Appendix A 

SISI Standards and Listing of Abbreviated Indicators 

A full explanation of the Standards and Indicators for School Improvement can be found at the following link 
http://www.education.ky.gov/KDE/Administrative+Resources/School+Improvement/Standards+and+Indicators+for 

+School+Improvement 

Standard 1 

The school develops and implements a curriculum that is rigorous, intentional, and aligned to state and local 
standards. 

1.1a Aligned with academic expectation, core content, program of studies 
1.1b Discussions among schools regarding curriculum standards 
1.1c Discussions among schools to eliminate overlaps, close gaps 
l.ld Vertical communication w/focus on key transition points 
l.le Links to continuing education, life and career options 
1.If Process to monitor, evaluate and review curriculum 
l.lg Common academic core for all students 

Standard 2 

The school uses multiple evaluation and assessment strategies to continuously monitor and modify instruc¬ 
tion to meet student needs and support proficient student work. 

2.1a Classroom assessments are frequent, rigorous, aligned 

2.1b Teachers collaborate in design of assessment, aligned 

2.1c Students can articulate the expectations, know requirements 

2.Id Test scores used to identify gaps 

2.1e Multiple assessments provide feedback on learning 

2.If Performance standards communicated and observable 

2.1g CATS coordination—building and district 

2.1h Student work analyzed 

Standard 3 

The school’s instructional program actively engages all students by using effective, varied and research-based 
practices to improve student academic performance. 

3.1a Varied instructional strategies used in all classrooms 

3.1b Instructional strategies/activities aligned with goals 

3.1c Strategies monitored/aligned to address learning styles 

3.Id Teachers demonstrate content knowledge 

3.1e Teachers incorporate technology in classrooms 

3.If Sufficient resources available 

3.1g Teacher collaboration to review student work 

3.1h Homework is frequent, monitored and tied to instructional practice 

Standard 4 

The school/district functions as an effective learning community and supports a climate conducive to per¬ 
formance excellence. 

4.1a Leadership support for safe, orderly environment 
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4.1b Leadership beliefs and practices for high achievement 

4.1c Teacher beliefs and practices for high achievement 

4.Id Teachers and non-teaching staff involved in decision making 

4.1e Teachers accept their role in student success/failure 

4.If Effective assignment and use of staff strengths 

4.1g Teachers communicate student progress with parents 

4.1h Teachers care about kids and inspire their best efforts 

4.11 Multiple communication strategies used to disseminate info 
4. lj Student achievement valued and publicly celebrated 

4.1k Equity and diversity valued and supported 

Standard 5 

The school/district works with families and community groups to remove barriers to learning in an effort to 
meet the intellectual, social, career, and developmental needs of students. 

5.1a Families and communities active partners 

5.1b All students have access to all curriculum 

5.1c School provides organizational structure 

5.Id Student instructional assistance outside of classroom 

5.1e Accurate student record keeping system 

Standard 6 

The school/district provides research-based, results driven professional development opportunities for staff 
and implements performance evaluation procedures in order to improve teaching and learning. 

6.1a Long term professional growth plans 

6.1b Building capacity with on-going PD 

6.1c Staff development aligned with student performance goals 

6.Id School improvement goals connected to student learning goals 

6.1e PD ongoing and job embedded 

6.If PD aligned to analysis of test data 

6.2a School has clearly defined evaluation process 

6.2b Leadership provides sufficient PD resources 

6.2c Evaluations and growth plans effectively used 

6.2d Evaluation process meets or exceeds statutes 

6.2e Instructional leadership needs addressed 

6.2f Leadership provides evaluation follow-up and support 

Standard 7 

School/district instructional decisions focus on support for teaching and learning, organizational direction, 
high performance expectations, creating a learning culture, and developing leadership capacity. 

7.1a Leadership developed shared vision 

7.1b Leadership decisions are collaborative, data driven, performance 

7.1c Leadership personal PD plan focused on effective skills 

7.Id Leadership disaggregates data 

7.1e Leadership provides access to curriculum and data 

7.If Leadership maximizes time effectiveness 

7.1g Leadership provides resources, monitors progress, removes barriers to learning 
7.1h Leadership ensures safe and effective learning 

7.11 Leadership ensures necessary SBDM policies 
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7.lj SBDM has intentional focus on student academic performance 

7.1k Leader has skills in academic performance, learning environment, efficiency 

Standard 8 

The organization of the school/district maximizes use of time, all available space and other resources to 
maximize teaching and learning and support high student and staff performances. 

8.1a Maximizes organization and resources for achievement 

8.1b Master schedule provides all students access 

8.1c Staffing based on student needs 

8.Id Staff's efficient use of time to maximize learning 

8.1e Team vertical and horizontal planning focused on improvement plan 

8.If Schedule aligned with student learning needs 

8.2a Resources used, equitable 

8.2b Discretionary funds allocated on data based needs 
8.2c Funds aligned with CP goals 

8.2d State/Federal funds allocated with CP goals and data needs 
Standard 9 

The school/district develops, implements and evaluates a comprehensive school improvement plan that com¬ 
municates a clear purpose, direction and action plan focused on teaching and learning. 

9.1a Collaborative process 

9.2a Planning process involves collecting, managing and analyzing data 
9.2b Uses data for school improvement planning 

9.3a Plans reflect research /expectations for learning and are reviewed by team 

9.3b Staff analyzes student learning needs 

9.3c Desired learning results are defined 

9.4a Data used to determine strengths and limitations 

9.4b School goals are defined 

9.5a School improvement action steps aligned with goals and objectives 
9.5b Plan identifies resources, timelines & person responsible 
9.5c Process to effectively evaluate plan 

9.5d Plan aligned with mission, beliefs, school profile, desired results 

9.6a Plan implemented as developed 

9.6b Evaluate degree of student learning set by plan 

9.6c Evaluate student performance according to plan 

9.6d Evidence to sustain the commitment to continuous improvement 
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